How to use xpath in Scrapy?
How to use xpath in Scrapy?
1 answer
by
sabryna ,
a year ago
@jayson.maggio
To use XPath in Scrapy, you can use the xpath() method provided by Scrapy's Selector class to select elements from an HTML or XML document.
Here's a simple example of how to use xpath() to extract the text content of all the p elements in an HTML document:
1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 |
import scrapy from scrapy.selector import Selector html = '<html><body><p>Hello World!</p><p>This is a test</p></body></html>' # Use the Selector class to parse the HTML selector = Selector(text=html) # Use the xpath() method to select all the p elements p_elements = selector.xpath('//p') # Extract the text content of the p elements for p in p_elements: text = p.xpath('text()').get() print(text) |
This will output the following:
1 2 |
Hello World! This is a test |
You can also use XPath expressions to select specific elements based on their attributes or the structure of the document. For example, the following XPath expression will select all the a elements with a href attribute that starts with http:
1
|
a_elements = selector.xpath('//a[starts-with(@href, "http")]') |
For more information on XPath and how to use it in Scrapy, you can refer to the Scrapy documentation and the XPath syntax reference.
15